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Objective 

The objective of this research was to explore the relationship between information access and 
design solution quality of high school students presented with an engineering design problem. 
This objective is encompassed in the research question driving this inquiry: How does 
information access impact the design process? This question has emerged in the context of an 
exploratory DR-K12 grant project titled, Exploring Engineering Design Knowing and Thinking 
as an Innovation in STEM Learning. The research work presented here has expanded the data set 
developed in the DR-K12 and examined the larger data set with a focus on how information 
access impacts design thinking. The opportunity to explore the impact of information gathering 
was not afforded in the DR-K12, but emerged as an area of interest during the pilot phase. 

Problem Statement 

The mission of the National Center for Engineering and Technology Education is to “improve 
understanding of the learning and teaching of high school students and teachers as they apply 
engineering design processes to technological problems” (National Center for Engineering and 
Technology Education, 2010). A common theme in the Center’s work in teaching engineering 
design has been the use of design challenges (Asunda & Hill, 2008; Becker, 2006; Becker & 
Custer, 2005, 2006; Cullum, Hailey, Householder, Merrill, & Dorward, 2008; Merrill, Childress, 
Rhodes, & R., 2006; Merrill, Custer, Daugherty, Westrick, & Zeng, 2007; Shumway, Berrett, 
Swapp, Erekson, & Terry, 2007; Tufenkjian & Lipton, 2007). The Center’s design process 
(Childress & Maurizio, 2007) is congruent with other models of engineering design (Dym & 
Little, 2004; Eide, Jenison, Mashaw, & Northup, 1998; Eide, Jenison, Northup, & Mickelson, 
2008; P. Moore, Atman, Bursic, Shuman, & Gottfried, 1995) and requires students to actively 
explore problem and solution space. 

This research proposal aligned with the Center’s goal of providing guidance for the field of 
Engineering and Technology Education on the development, implementation, and evaluation of 
engineering design. In the Center’s model, students are expected to perform the following 
processes (Childress & Maurizio, 2007, p. 3): 

1 . Identification of a need 

2. Definition of the problem/specifications 

3. Search 

4. Develop designs 

5. Analysis 

6. Decision 

7. Test prototype and verify the solution 

8. Communication 



As students work through these iterative stages of the design process, a need for information 
arises. High school students are novice designers, these students have a limited background and 
limited experiences in design thinking. This limited experiential background makes gathering 
information even more critical for their success. Students might be gathering information about 
the problem and its context including cultural, environmental, geographic types of information. 
As students begin to think about their potential solutions, they may search for information about 
previous solutions or current solutions that are insufficient in an attempt to benchmark. 
Developing their design may include information about standard materials and their properties or 
readymade components that could be integrated in a novel way. Analysis may leverage 
identifying variables and their relationships to be used in predicting performance. These 
variables and relationships might not be heuristics familiar to students or may be too complex to 
memorize which pushes the student to search for information. 

The demand for information beyond the immediate identified need is substantial and ubiquitous 
in the design process. Ennis and Gyeszly (1991) found that gathering information was as 
essential element of the expert designers’ approach to problem solving and that generation of 
ideas was influenced by the information. Experts have practice accessing information and are 
familiar with the structure and content of databases, previous project examples and other experts 
with whom to collaborate. Novice students do not have these engineering domain specific 
information literacy skills. In a recent study comparing college student and expert engineering 
design behaviors, Atman et al. (2007) stated that “Results support the argument that problem 
scoping and information gathering are major differences between advanced engineers and 
students, and important competencies for engineering students to develop” (p. 359). To facilitate 
a successful learning environment during implementation of the engineering design challenge, 
students need access to information. Teachers can provide information to students relevant to the 
challenge at hand through discussions or print resources. An alternative or supplement to the 
teacher’s resources could be providing students with access to the internet. Today’s young 
people are digital natives (Prensky, 2009) and have grown up with information access via 
multiple channels, and thus, internet access would appear to be ecologically appropriate. Prensky 
(2009) argued that digital access to information and analytical tools enhance our thinking power. 

Theoretical Foundation 

The foundation of this study was built on similar previous studies of college students. Design 
problems in these previous studies are ill- structured and open-ended. These kinds of problems 
have many potential solution paths stemming from a need or problem. The Carnegie Eoundation 
for the Advancement of Teaching has prepared a series of studies including a focus on educating 
engineers (Sheppard, Macatangay, Colby, & Sullivan, 2009). This research identified reflective 
judgment as an appropriate framework for understanding the cognitive development of design 
thinking. “As individuals develop mature reflective judgment, their epistemological assumptions 
and their ability to evaluate knowledge claims and evidence and to justify their claims and 
beliefs change” (Sheppard, et al., 2009, p. 25). 



King and Kitchener (1994) have identified seven stages of reflective thinking organized into 
three clusters: pre-reflective thinking, quasi-reflective thinking and reflective thinking. Results of 
a ten-year longitudinal study of reflective judgment (King, 1977; Kitchener, 1977-78; Kitchener 
& King, 1981) suggested that juniors in high school have a cognitive development that tended to 
approach stage 3 while college juniors tended to be nearing stage 4. This indicated that on 
average high school students are in the pre-reflective thinking cluster while college students are 
in the quasi-reflective cluster of development. Results of design thinking studies conducted on 
the college level might yield different results based on the advanced cognitive development of 
college students. The quasi-reflective cluster of development is characterized by people 
recognizing that some problems are ill- structured and that uncertainty requires judgment. This 
quasi-reflective cluster differs from the pre-reflective thinking cluster wherein individuals 
perceive knowledge to be certain and its sources are that of authority or direct experience. These 
developmental differences in cognitive approach to ill- structured problems suggest that high 
school student performance may differ from college student and expertise performance. This 
framework for cognitive development also suggested that high school students may have a 
tendency to search for information about other peoples’ solutions (an authority on playground 
design) rather than internalize they are the designer of this solution. By providing access to the 
internet, images and descriptions of other solutions are at easy reach and may alter the decision 
making process. 

Teacher provided information can be well focused and therefore reduce the time students spend 
searching. Christiaans and Dorst (1992) discovered that some students get “stuck” gathering 
information and this fixation prevents students from making progress on their design. On the 
other hand, the teacher would be limiting student creativity by providing information that is bias 
toward a solution or set of solutions envisioned by the teacher. Preparing all the possible 
information for students would be a demanding (perhaps impossible) task and, based on limited 
preparation time, will have to be abbreviated. If the teacher allows students to utilize the internet 
for web based searches, a virtually unlimited pool of information is accessible. Successful 
negotiation of this material requires complex information literacy skills and time management. 
Students could spend countless hours researching online, possibly drifting aimlessly, rather than 
thinking critically about the design challenge at hand. 

Design Thinking 

The discrepancy between our society’s reliance and dependence on technology and our ability to 
understand various technological issues has emerged as a serious concern for educators. 
“Technology is the outcome of engineering; it is rare that science translates directly into 
technology, just as it is not true that engineering is just applied science” (National Academy of 
Engineering, 2004, p. 7). Specifically, “Americans are poorly equipped to recognize, let alone 
ponder or address, the challenges technology poses or the problems it could solve” (Pearson & 
Young, 2002, pp. 1-2). The relationship between understanding engineering and technological 
literacy is of special urgency during the high school years, since “technologically literate people 



should also know something about the engineering design proeess” (Pearson & Young, 2002, p. 
18). 


Design thinking is fundamental to understanding the technologically dependent nature of our 
society. A need for a technologically literate populace, therefore, includes an understanding of 
the engineering design process. It is this design process which connects technology and 
engineering, two elements of STEM education. “Design is the central component of the practice 
of engineering and a key element in technology education” (Pearson & Young, 2002, p. 58). 
Sheppard, Macatangay, Colby, and Sullivan (2009) stated that “engineering design involves a 
way of thinking that is increasingly referred to as design thinking: a high level of creativity and 
mental discipline as the engineer tries to discover the heart of the problem and explore beyond 
the solutions at easy reach” (p. 100). This study will identify quality high school engineering 
learning and teaching environments in a criterion based sampling strategy, the setting envisioned 
by Pearson and Young (2002), where “technology teachers with a good understanding of science 
and the interactions between technology, science, and society will be well prepared to work with 
other teachers to integrate technology with other subjects” (p. 108). 

Operationalizing the Construct of Design Thinking 

While design thinking is an elusive and difficult construct to define, measurements for this study 
were a pertinent subset of measurements consistent with previous literature, much of which was 
generated through work of the Center for Engineering Learning and Teaching (Atman, Chimka, 
Bursic, & Nachtmann, 1999; Atman, Kilgore, & McKenna, 2008; Morozov, Yasuhara, Kilgore, 
& Atman, 2008; Mosborg et ah, 2005; Mosborg et ah, 2006b). In this study, measurements 
included: 

Solution quality 

Time spent developing a solution 

Time allocated to gathering information 

Number of information requests 

Identification of categories of information requested 

Methodology 

A quasi-experimental design was implemented where two groups of students were identified. 
One group of students had internet access during the design session while the other group did 
not. All other variables and student demographics were held constant to the best of the research 
team’s abilities. Students were presented with a design problem and provided three hours to 
develop a solution while “thinking aloud”. 

Sample 

A sample size of sixty was used in this study, which yielded thirty students per group. Students 
volunteered from six schools spanning four states. The schools selected to participate had a 



recognized engineering program associated with an outreach effort by a university engineering 
program. Four universities were involved in identifying the six target schools. Curricular 
offerings at the high school included Project Lead the Way, EPICS High and locally developed 
courses supported by their regional University. 

Teachers at the target schools permitted advertising to their students. Students in this study were 
representative of experienced students having taken most or all engineering related courses at 
their high school. Students were recruited who were actively engaged in the study of engineering 
design through a criterion sampling strategy (Creswell, 1998) using the following criteria: 

• The high schools will have an established program of study which employs a focus on 
engineering in a sequence of courses developed in association with an engineering 
outreach effort as part of a university program. 

• In these courses, students will participate in design activities which engage their critical 
thinking and problem solving skills within the framework of the engineering design 
process. 

• Students will be selected who represent diverse backgrounds and have chosen to enroll in 
this sequence of courses. 

Instrumentation 

The playground problem has been used in multiple studies and can be traced to Dally and Zang 
(1993) They identified the need for project driven approaches in the freshman engineering design 
course to increase student performance and retention Project driven approaches situate student 
learning of abstract concepts through real world applications in an experiential activity. In the 
original activity, students designed a swing set with slides and seesaw. Atman, Chimka, Bursic 
and Nachtmann (1999) revised the foundational work of Dally and Zang to create a playground 
design problem. In this challenge, engineering students were presented with a brief playground 
design task and access to background information upon request. Participants were provided with 
a maximum of three-hours to develop a solution to the problem while thinking aloud. Mosborg, 
Adams, Kim, Atman, Turns, and Cardella (2005) applied the playground design challenge using 
the “think aloud” research protocol to 19 practicing engineers who were identified as experts in 
the field. Mosborg, Cardella, Saleem, Atman, Adams and Turns (2006b) compared groups of 
freshman and senior engineering students with practicing engineers using data previously 
collected on the playground design challenge. Atman, Kilgore and McKenna (2008) analyzed 
data from previous studies using a lens focused on the language of design and its relationship to 
design thinking as a mediator and how this internalization of design thinking relates to language 
acquisition. This work provided a well developed design task and data for comparisons with the 
proposed study participants. 



The playground design task is an effective task to demonstrate design thinking by students as it is 
an open-ended, realistic, accessible, and complex problem (Mosborg, et ah, 2006b). The 
endeavor to model problem solving satisfactorily has eluded scholars across domains (Hayes, 
1989; Newell & Simon, 1972; Polya, 1945; Rubenzer, 1979). Engineering design problems in 
practice tend to be structurally open-ended and highly complex. An open-ended problem may 
have numerous solution paths and be bound by some rigid and some negotiable constraints, not 
always presented with the problem. Engineering design is more than the mere manipulation of 
numbers and the solving of scientific equations. The processes employed in engineering design 
encompass a broad variety of topics and field of study. Through the lens of an ethnographer, 
Bucciarelli (1988) described engineering as a social process. The National Academy of 
Engineering suggested that engineering education was deficient if it did not include the global 
perspective in engineering design such as social, political, and environmental issues (2004, 

2005). The global perspective of engineering is synonymous with the term “systems 
engineering.” Systems engineering involves design from the whole systems level rather than 
from an isolated modular perspective. 

Not only do open-ended problems more accurately reflect industry practices, they also provide 
students more flexibility and choice (Mawson, 2003). As students are given more freedom and 
choice, they become further engaged in their own education (McKeachie, 2006; Schulz, 1991). 
Authentic problems provide a broad impact, rich in real-world contexts. As such, open-ended 
problems give the student an authentic experience and greater motivation (Yair, 2000). 
Eurthermore, playgrounds are familiar to students as they are common to most neighborhoods. 
This design activity does not require domain-specific knowledge such as electrical, biological, or 
mechanical engineering and, therefore, is accessible to many student participants with a variety 
of backgrounds and experiences (Mosborg, et ah, 2006b). 

The participants of the playground problem were given a one page design brief. The constraints 
were vague with the participant, acting as an engineer, assigned to design a playground on a 
donated city block. The constraints included limited budget, child safety, and compliance with 
laws or zoning. The participant was also able to query the research administrator for additional 
specific information such as the lot layout, cost of materials, or neighborhood demographics. 
There was a three-hour time limit for completion of the design proposal. The participants 
presented a written proposal describing their design. This activity engaged the participants in 
problem framing and developing an initial solution. Eimitations of this design task included the 
lack of opportunity for participants to investigate the need for a solution; as it was directly 
presented to them. Students did not have an opportunity to construct physical models or 
prototypes. Participants were aware that implementation of the design project would not occur, 
and their designs would not have the potential to become realized. 



Administration of the Design Problem 

A team of graduate and undergraduate students and university faculty conducted the data 
collection. A total of six students were involved with data collection efforts spanning just over 
one year. A lead researcher administered the problem and trained the student researchers through 
observation and direct participation. The researcher reviewed data collected and reflected with 
the students following the session. Student researchers’ administration of the design problem was 
video recorded as part of the data collection and videos were reviewed for training purposes to 
ensure consistency while data collection was active. 

Data included video and audio recorded design sessions. Video cameras were small, pocket sized 
on miniature tripods to minimize the invasive intrusion. Audio recorders were used as a backup 
to the video cameras and were positioned on the work space near the student. Generally, students 
were wired with a lavaliere microphone to ensure high quality audio feeds. Students generated 
documents and other artifacts with traditional office supplies provided. Artifacts typically 
included sketches, notes, formal drawings and, in one case, a prototype solution made from torn 
and folded sticky notes. Two-dimensional works were anticipated by the research team and 
scanned to digital image form. The three-dimensional work was video recorded. Data were 
archived in digital format on hard drives. Drives were shared with the research team for analysis 
purposes. 

Data Analysis 

The playground problem coding scheme was congruent with the approach used in prior studies 
(Atman, et ah, 1999; Bursic & Atman, 1997; Mosborg, et ah, 2005; Mosborg, et ah, 2006b). 
Solution quality, time and information requested were considered. 

Solution Quality 

Data gathered from the student participants was analyzed for solution quality. Solution quality 
was identified in previous work as being an essential measure of the design process. In the 
previous work, the design session concluded with the designer presenting the administrator with 
their final design. This final design was assessed based on how well it met the design criteria. 
Consistent with the previous work (Atman, et ah, 1999; R. C. Moore, Goltsman, & lacofano, 
1992), this study used a four part measurement to evaluate design solution quality: 

1 . The first element included seven criteria based on the constraints provided in the 
problem brief. 

2. Moore, Goltsman and lacofano (1992) published documentation used to assess the 
safety of playground designs including 33 criteria were appropriate for all 
playgrounds. 

3. According to Moore, et al. (1992), an additional 79 criteria were specific to elements 
which could potentially be included in a playground. These criteria applied only if the 
element was included in the design. Therefore a participant would receive a score 
based on the elements they included and the level of appropriateness per element. As 



an example, wood should be protected from rot, but only if wood was included in the 
design. A student’s design score would be lower if they included wood but did not 
consider longevity of the material. 

4. The fourth element of quality included a Likert scale rubric relating of five 

categories: diversity of activities, aesthetics, protection from injury, uniqueness, and 
technical feasibility. The rubric was adopted from Atman’s (1999) and was specific to 
elements of the designer’s solution and standards for playground design. 

A spreadsheet was created with each of the previous four quality elements. A pair of 
undergraduate students assessed the quality of the solutions and facilitated inter-rater reliability 
analysis. The criteria and safety considerations (elements 1, 2 and 3 above) included 119 points 
which were awarded on a binary scale, 0 if evidence was not presented in the data or 1 if 
evidence was presented that the student’s design included this consideration. The fourth element 
(rubric) included a five point rating for each of the categories. A total quality score was 
presented by Atman (1999) which was calculated by averaging elements 2, 3, and 4 together. 
Each of these elements was weighted equally and the three were averaged to yield a total quality 
score, presented in this document as a comparison to previous work. A more full description of 
these elements and their assessment was described by Mosborg, et al. (2006b). 

Solution quality measurements were coded independently by two undergraduate research 
assistants. Each research assistant had been involved with the data collection and, as a result, was 
familiar with the design task. Training and calibrating the quality and request coding was done 
by introducing the research assistants to the coding scheme. A few participants’ data sets were 
reviewed together and coded collaboratively as a team. The research assistants coded one data 
set independently and inter-rater reliability was calculated. Discrepancies were discussed and 
resolved. Assistants coded the next data set and compared results. This iterative process 
continued until reliability values were satisfactory (about four iterations). At that point, 
previously coded data was re-coded and all remaining data was coded. Each assistant was 
assigned 30 participants for a total of sixty in this study. 

Eleven of sixty participants’ quality and information request data were coded by both research 
assistants. These data were used to calculate inter-rater reliability. Reliability was calculated as 
follows for the dimensions of solution quality: 

Constraints: Kappa value 0.709 
Playground Safety Criteria: Kappa value 0.821 
Rubric Rating: Cronbach’s Alpha value 0.880 



Time 

Time is a limited resource and how designers allocate their time in areas of the design process 
has been a focus of previous work. Two measurements of time were made while the designers 
are at work: 

1 . Total time engaged in design was measured from start to finish and was limited 
to three hours. 

2. Time allocated to information gathering was measured. In previous work, time 
spent gathering information was critical to generating quality solutions, 
however, becoming stuck in the information gathering mode became 
detrimental. This measurement was defined as time participants spent searching 
for information, reviewing information requested and considering information 
they would request (Mosborg et ah, 2006a). This time did not include reading 
the problem statement. 

Data were independently analyzed for time by two coders. Each coder was responsible for 
approximately 30 participants. While pairs of coders were compared on this data set, analysis 
spanned a duration of one year and involved three different pairs of raters. Training was 
conducted by introducing the coders to the coding scheme as outlined by Mosborg, et al. 

(2006a). Training on coding was done in an iterative fashion where each participants’ data were 
coded independently and compared. Coders then negotiated to consensus and documented their 
improved understanding. This iterative calibration went through multiple cycles before Kappa 
values were satisfactory. One pair of raters reviewed 16 participants, overlapping for comparison 
purposes on 25% of the time with a Kappa value of 0.809. A third coder was hired and compared 
to one of the previous coders. The Kappa value for this comparison was 0.939, representing 25% 
of the data set and the third coder coded approximately 14 participants. A fourth coder was hired 
and calibrated with the third for the remaining 30 participants. Their average Kappa value on 
25% of the data set was 0.950. 

Information Requests 

The data were coded for “gathering information” as presented by Mosborg et al., (2006b, p. 15). 
The gathering information element of the design process was one of nine elements considered in 
the previous foundational work and included students looking for information to help them solve 
the problem. Coding included what information was requested by the participant and at what 
point in time. Also consistent with prior research, the following categories of information were 
available for participant request (Mosborg, et al., 2006b, p. 21): budget, information about the 
area, material costs, neighborhood opinions, utilities, neighborhood demographics, safety, 
maintenance concerns, labor availability and costs, legal liability, material specification, 
supervision concerns, availability of materials, body dimensions, disabled accessibility, technical 
references, and other information. 



Within these categories, specific detailed information were available upon request, and 
participants’ requests for this information provided researchers with data regarding problem 
scoping and definition techniques employed. “Question asking while designing is influential to 
the cognition of designers. It is related to the cognitive aspects of their problem solving, 
creativity, decision making, and learning processes, and, consequently, to their overall 
performance” (Eris, 2004, p. 1 1). In addition to paper based information available upon request, 
the internet group had a Google search browser launched on a laptop computer at their 
worktable. Participant information gathering behaviors were coded regardless of source (digital 
or paper based) for each group. Adopted from previous literature, (Mosborg, et ah, 2006a, pp. 
11-14), information requests were coded into the following categories, with one exception added 
by this research team, which was “Image Searches”: 

1. AGE - Statements addressing the “1-10 years of age” constraint. 

2. OCCUPANCY - Statements addressing the “12 children kept busy” constraint. 

3. ACTIVITIES - Statements addressing the “at least 3 activities” constraint. 

4. SAEETY - Statements addressing the “safe for children” constraint. 

5. HANDICAPPED ACCESSIBIEITY - Statements addressing safety or accessibility for 
persons with disabilities. 

6. SUPPEIER - Statements addressing the “use material available at any hardware or 
lumber store” constraint. 

7. SCHEDUEE - Statements addressing the “ready in 2 months” constraint for constructing 
the playground 

8. equipment. 

9. CEARITY - Statements addressing the “explain your solution as clearly and completely 
as possible” 

10. constraint. Includes statements about making instructions or diagrams for the people 
building the playground. 

11. BUDGET - Statements about the amount of money available for the project. 

12. MATERIAE COST - Statements about the cost of specific materials. 

13. MATERIAE COST and BUDGET - Statements about the cost of specific materials with 
respect to budget or affordability. 

14. EABOR - Statements about workers for the project. 

15. MATERIAE TYPE - Statements about the general type of material needed (e.g., wood, 
2x4’s, steel, screws, nails, paint). 

16. MATERIAE SPECIEICATIONS - Statements about technical material requirements. 

17. TECHNIC AE REEERENCE - Statements about technical construction requirements. 

18. DIMENSIONS - Statements about the specific measurements (typical, ballpark, or 
actual) of playground 

19. equipment, layout, or the lot. 

20. BODY DIMENSIONS - Statements about human body size(s). 

21. NEIGHBORHOOD AREA - Statements about the location of objects in the area 
surrounding the lot. 

22. DEMOGRAPHICS - Statements about the composition of the neighborhood population. 



23. OPINIONS - Statements about stakeholders’ reactions to the proposed playground, or 
their preferences for equipment or activities. 

24. NEIGHBORHOOD CONDITIONS - Statements about other conditions of the area. 

25. PARK AREA INSIDE THE EOT - Statements about the lot’s characteristics or layout. 

26. UTIEITIES - Statements about gas, water, or power lines. 

27. EACIEITIES - Statements about playground facilities such as bathrooms, night lighting, 
or water fountains. 

28. MAINTENANCE - Statements about property or equipment maintenance for the 
playground’s operation. 

29. EEGAE - Statements about liability for potential injuries or accidents. 

30. SUPERVISION - Statements about looking after children during playground hours. 

3 1 . IMAGE SEARCHES - (introduced by this research team) - related to students searching 
for pictures of playgrounds or related topics to look at. 

Information requests data were assembled into a spreadsheet which included the request made by 
the participant and time of request. When the request was verbal to the administrator, the 
information requested was documented by the administrator. When the request was made via 
computer on the internet, the search term and resulting websites visited were recorded by a 
software application running in the background called Spector Pro. Each undergraduate coder 
was responsible for 30 participants. Eleven of the sixty were reviewed by both coders to permit 
inter-rater reliability to be computed. Coders were instructed to document purposeful information 
requests. They attempted to count only unique information requests which meant they had to 
segment multiple requests together in some instances and code as one request. Eor example, if a 
student was searching for the cost of a 2x4 and searched an online retailer, a second online 
retailer, a classified advertisement online and requested this information from the administrator, 
the individual actions were grouped and counted as one information request about the cost of a 
2x4. Coders reviewed the spreadsheet of information requests and watched the video to code the 
data situated in context. Spreadsheets from each coder were aligned for training purposes and 
compared line by line. Coders tallied total requests for each of 31 categories. Total requests are 
reported here and were used to calculate inter-rater reliability with a Cronbach’s Alpha: 0.850. 

Results 

Data were coded for this study and assembled into a spreadsheet for quantitative analysis. SPSS 
version 18 software was used to generate mean and standard deviation data comparing the two 
groups of participants (one with internet access and one without). Independent samples T-Tests 
were used to compare mean scores for each of the comparisons. A Levee’s Test of Equality of 
Variances was used to determine if T-Tests for equality of variances should be assumed or not. 
Eour tables are presented here relative to the research question and, where appropriate, draw a 
comparison to Atman’s (2007) work and collegiate freshman data. 



Solution Quality 

Solution quality included four measurements: Constraints met, safety criteria for all playgrounds, 
design element specific safety criteria, and rubric ratings. Of the seven constraints presented in 
the design brief, both groups averaged meeting just under 4 of the 7 with no statistical difference 
between group means. This is, on average, 1 constraint less than college freshmen. The overall 
safety criteria score was generated by adding the criteria for all playgrounds with the element 
specific criteria and dividing by the total possible criteria score (with each designer’s specific 
elements) to get a score out of 1 . The internet access group had a slightly higher mean score, but 
no significant differences existed between groups. The rubrics measured five categories: 
diversity of activities, aesthetics, protection from injury, uniqueness, and technical feasibility. 
The rubric scores were averaged per participant and compared. Students without internet access 
had slightly, but not statistically significantly higher mean scores. The quality score included an 
average of design criteria scores and rubric scores, weighted equally, and presented out of 1. The 
no internet access group scored slightly higher, but differences were not statistically significant. 
Both high school groups scored lower than college freshman with approximately 0.30 as 
compared to 0.45. Solution quality indices are presented in Table 1 as mean scores with standard 
deviation in parenthesis and independent samples T-test significance values are shown. 

Table 1. Solution Quality 


High School College 

Freshman 


Internet 

No Internet 

T-test (2 tailed) 

Atman (2007) 

Access 

Access 

Significance 

n=26 

Group n=30 

Group n=30 




Constraint Score (out of 7) 

3.967 (1.63) 

3.967 (1.19) 

1.000 

5.0(1.!) 

Safety Criteria for all 
playgrounds (out of 1) 

0.222 (0.09) 

0.224 (0.07) 

Not tested 


Design element specific 
safety criteria (out of 1) 

0.132 (0.13) 

0.0924 

(0.07) 

Not tested 


Overall safety criteria score 
(out of 1) 

0.182 (0.07) 

0.170 (0.05) 

0.422 


Rubric (out of 5) 

2.640 (0.83) 

2.920 (0.93) 

0.222 


Quality score (Design 

0.294 

0.300 

Not Tested 

0.45 (0.1) 


criteria scores and rubric, 
weighted equally - out of 1) 

Note: Standard deviations are in parenthesis. 



Time 


Two measurements are presented related to time: Total design session duration and time 
allocated to information gathering. Students with access to the internet spent significantly more 
time engaged in the design process with a mean of 139 minutes as compared to students without 
internet access whose mean was 90 minutes. Most of the difference in design session duration 
was explained by the additional time allocated to gathering information. The internet access 
group allocated significantly more time (mean of 42 minutes) to gathering information as 
compared to the group without internet access (mean of 10 minutes). The mean non-internet 
access group was higher than the college freshmen but less than high school students with 
internet access. Table 2 presents mean values for the number of minutes engaged in the design 
session and information gathering, standard deviation shown in parenthesis, significance test 
probability values and a comparison to college freshman. 

Table 2. Minutes Spent Designing and Allocated to Information Gathering 



High School 


College 

Freshman 

Internet 

No Internet 

T-test (2 

Atman (2007) 

Access Group 

Access Group 

tailed) 

n=26 

n=30 

n=I8 

Significance 



Design Session Duration 

139 (44) 

90 (56) 

0.002* 

104 (41.0) 

Information Gathering 

42 (22) 

10(13) 

<0.001* 

13.8 (12.6) 


Note: * Denotes statistically significant difference at the 0.05 level. Standard deviations 
are in parenthesis. Also note that at the time of this publication, only 18 of the 30 students 
times were analyzed. A revision is pending will be replacing this document soon. 


Information Requests 

Both groups had access to paper based information by requesting it specifically from the 
administrator but only one group had internet access. Each request for information was 
documented and the groups were compared. The group with internet access requested 
significantly more information with a mean score of 16.7 pieces of information as compared to 
the groups without internet access with a mean of 3.7 pieces of information per student. Students 
with internet access requested more different kinds of information measured by spanning 
multiple categories. Thirty-two categories were identified including a category of “other”. An 
information request fell into the “other” category when it did not fit into the previous 3 1 
categories or the data analyst could not figure out which category was most appropriate because 
the request was vague. The group without internet access was most comparable to the college 
freshman, but they requested less information spanning fewer categories. Table 3 displays mean 



numbers of information requests and eategories spanned, significance values for the independent 
samples T-tests and a comparison to collegiate freshman. 


Table 3. Information Requests and Information Categories 



High School 


College 

Freshman 

Internet 

No Internet 

T-test (2 

Atman (2007) 

Access Group 

Access Group 

tailed) 

n=26 

n=30 

n=30 

Significance 



Information Requests 

16.7 (8.7) 

3.7 (5.0) 

<0.001* 

11.4(10.1) 

Categories 

7.0 (2.6) 

2.0 (2.2) 

<0.001* 

3.4 (2.9) 


Note: Standard deviations are in parenthesis. * Denotes statistically significant difference 
at the 0.05 level. 


Information requests were categorized and the mean number of requests per student was 
identified for each category. Table 4 presents the mean and standard deviation data for both 
groups in this study for each category. Statistically significant differences are considered at an 
alpha of 0.05. Material costs were requested more than any other category of information for 
both groups and significantly more among the internet access group compared to the no internet 
access group. 


Table 4. Mean Requested Pieces of Information per Student by Category: 



Internet 

No Internet 

T-test (2 


Access Group 

Access Group 

tailed) 


n=30 

n=30 

Significance 

Material Cost 

6.57 (5.1) 

1.67 (3.0) 

<0.001* 

Other 

1.33 (1.4) 

0.33 (0.8) 

0.001* 

Activities 

1.30(1.4) 

0.00 (0.0) 

<0.001* 

Dimensions 

1.27 (I. I) 

0.07 (0.4) 

<0.001* 

Disabled Accessibility 

TOO (0.5) 

0.60 (0.6) 

0.006* 

Material Type 

0.87 (1.8) 

0.07 (0.3) 

0.023* 

Image Searches 

0.80(1.2) 

0.00 (0.0) 

0.001* 

Material Specifications 

0.63 (0.8) 

0.10(0.4) 

0.002* 

Budget 

0.57 (0.6) 

0.23 (0.4) 

0.013* 

Safety 

0.53 (1.0) 

0.13 (0.4) 

0.046* 

Technical Reference 

0.37 (0.7) 

0.03 (0.2) 

0.019* 

Neighborhood Area 

0.27 (0.6) 

0.10(0.4) 

0.233 



Demographics 

0.17 (0.7) 

0.07 (0.4) 

0.464 

Body Dimensions 

0.17 (0.4) 

0.07 (0.3) 

0.235 

Clarity 

0.13 (0.4) 

0.00 (0.0) 

0.103 

Eabor 

0.13 (0.4) 

0.00 (0.0) 

0.103 

Opinions 

0.13 (0.4) 

0.10(0.4) 

0.732 

Neighborhood Conditions 

0.13 (0.4) 

0.00 (0.0) 

0.043* 

Maintenance 

0.10(0.4) 

0.00 (0.0) 

0.184 

Utilities 

0.07 (0.3) 

0.03 (0.2) 

0.561 

Supplier 

0.07 (0.3) 

0.03 (0.2) 

0.561 

Eacilities 

0.03 (0.2) 

0.00 (0.0) 

0.326 

Eegal 

0.03 (0.2) 

0.00 (0.0) 

0.326 

Age 

0.00 (0.0) 

0.00 (0.0) 

1.000 

Occupancy 

0.00 (0.0) 

0.00 (0.0) 

1.000 

Schedule 

0.00 (0.0) 

0.00 (0.0) 

1.000 

Material Cost and Budget 

0.00 (0.0) 

0.00 (0.0) 

1.000 

Park Area Inside the Eot 

0.00 (0.0) 

0.07 (0.3) 

0.161 

Supervision 

0.00 (0.0) 

0.00 (0.0) 

1.000 


Note: Standard deviations are in parenthesis. * Denotes statistieally signifieant differenee 
at the 0.05 level. 


Discussion and Implications 

Students provided with internet access used this resource. As measured by time, they spent 
significantly more time searching and they made significantly more information requests. The 
additional effort measured by time and requests did not impact solution quality. Solution quality 
was measured by number of constraints addressed, expert playground design criteria addressed 
and a series of rubric scores and there was no statistical or practical difference between the 
groups. 

Engineering design challenges provide a context for students to engage with the material, but are 
time consuming. These data suggest providing internet access is not an efficient tool in the 
design process. Students requested substantially more information regarding costs of materials 
than any other category. This pattern was consistent among both groups, but the number of 
requests were dramatically greater for the internet access group. While costs are an important 
consideration in the design process, they are not the only consideration and investigating cost did 
not have measurable impacts on the solution quality. 

Access to information via the internet is ubiquitous in the workplace and a rich resource. On the 
surface these data suggest internet access was a waste of time relative to its negligible impact 
solution quality. Students had little difficulty navigating the internet as a source of information, 
but it had no impact on their solution quality. Elements such as the end user of the solution. 



community where the solution is situated, transportation of materials to the site, communication 
with the builder about the solution, and maintenance were seldom considered. 


The implication of this work may be that students do not know what information is most helpful 
in developing a good quality solution. They have access to an unlimited information resource, 
but don’t take advantage of much more than material cost. One student of the sixty considered 
the constraints and asked for information: “What does this community consider safe?” While 
other students read the requirement for safety, it was only one student who treated this 
requirement as more than just a superficial check off. He recognized that safety was a continuum 
and playgrounds could be more or less safe, not just “safe” or “not safe”. Students frequently 
drew on previous experience, which is consistent with the work of King and Kitchener (1994). 
Student memories of personal experiences may have influenced their thinking and may have 
been the substantial source of information in their work but was unmeasured by this study. 

Engineering design should consider a systems level approach to solutions. Few students 
requested information about the social, political, environmental, geographic, or historic 
conditions related to the playground location. This lack of requests was perhaps a measure of a 
more significant concern about problem definition. Student understanding of the problem was 
not measured, but these data highlight a potential concern that students did not fully comprehend 
the problem at hand. Low quality solutions could be a result of a poor exploration of the 
problem. Problem definition was not measured as part of this study, but was considered in the 
pilot study for the DRK-12 project titled. Exploring Engineering Design Knowing and Thinking 
as an Innovation in STEM Learning (Mentzer, Becker, & Park, 201 1). According to their study, 
high school students spent 3.14 minutes, just over 1/3 of the time spend by experts. 

Students were asked to develop a solution that could be built without requiring the designer 
answer any questions. The extent to which a builder would be ready to order materials and begin 
construction was not quantitatively measured, but in a general qualitative consideration, few 
student designs could be built. Most were messy, disorganized, and incomplete. One student 
commented that she did not need to gather information about material strengths because the 
builder would follow her design during the construction phase. The builder would then test out 
the design by climbing on the playground equipment, if it broke, the builder would rebuild the 
equipment using stronger materials. If the equipment did not break, it would be ready for public 
use. This student, representative of more than one-half the participants, never asked for a budget. 
Questioned afterwards about how she met the constraint about “not costing too much” she 
confidently stated that all the materials for her design were available at the local hardware store, 
and “everybody knows nothing at the hardware store costs too much”. Her reflection suggested 
that she was thinking about costs and budget even though she had not requested this information 
directly. 



Future Research 

Consistent with previous literature, this study investigated the information requests of student 
designers. Immediately evident in the data analysis phase was an unplanned opportunity whieh 
the researeh team was not prepared to exploit. Students made requests for information, but in 
both groups (internet and no internet), students diseovered and used more and different 
information then they had initially been looking for. The eoneept of measuring information 
discoveries may be a value for future researeh as it impaets student thinking. As an example, one 
student initially searehed for pietures of playgrounds (eategorized as images searehers). In 
reviewing the pietures, she aeeidentally diseovered a need for maintenanee. The maintenanee 
doeument online was used to guide her material seleetions (she avoided wood because it might 
splinter). The maintenance information had related links to safety which lead her to consider fall 
heights based on equipment design. The allowable equipment height was related to surface 
materials and this pushed her to specify soft surface materials in target areas around equipment. 
The research team believed these were serendipitous discoveries that would not have been 
considered if they had not appeared on the computer screen as related links to the original search. 
This project did not permit a detailed investigation of information discoveries, but the research 
team noted a series of related discoveries based on a single information request as a consistent 
theme. It seems feasible to investigate information discoveries as a web of interconnected 
elements, similar to concept mapping in future work. 

Further research should investigate problem definition more deeply. Students’ understandings of 
the problem were questionable and could be the reason solution quality was low and additional 
information requests minimally impacted solution quality. While previous work investigated the 
amount of time a student spent on the problem definition, perhaps future work could investigate 
the extent to which students understand the problem. Time is a proxy for effort spent considering 
the problem, but it may be un-calibrated in younger learners as they are not familiar with the full 
breadth of considerations appropriate to the engineering design process. Conducting formal 
interviews after the design process might provide an opportunity to provide insight on student 
understanding of the problem. 

Classroom observation data combined with an investigation of curriculum used in the classroom 
may provide insight as to the treatment of problem scoping in the design process. Students in this 
study may have limited experience in investigating the problem, these data were not collected. A 
formal curriculum review could lend insight to the procedures and methods expected of students 
as they engage in the design process relative to information gathering. Student may not be 
coached on how to identify relevant information while developing their solution. 

Summary 

Sixty students were presented with a design challenge. Thirty of these students had access to the 
internet and the other 30 did not. Both groups were asked to think aloud while attempting to 
develop a solution individually within a three hour time limit. Students represented six states 



geographically distributed across the U.S. Solution quality was measured consistent with 
previous literature. Students with access to the internet spent substantially more time developing 
a solution, but their solution was of similar quality to the group who did not have access to the 
internet. The group with internet access made more information requests in more varied 
categories and spent more time doing so. Further research should investigate a potential concern 
which emerged during the data analysis regarding student understanding of the problem. If 
students failed to fully understand what they were asked to do, they may have just drifted 
aimlessly through relevant information without knowing how to use it efficiently. 
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